Optimizing loading of a payload carrier of a machine

ABSTRACT

A method for loading a payload carrier of a machine includes receiving, from a camera on the machine, a two-dimensional image of an interior of the payload carrier as material is loaded into the payload carrier. The method further includes filtering the image to identify a contour of the loaded material and determining an area of the contour. The method further includes controlling a display device indicate the determined area.

TECHNICAL FIELD

The present disclosure is directed to machine production optimization,and more particularly, to production optimization for operation ofexcavating machines such as a wheel tractor scraper.

BACKGROUND

Earthmoving machines may be used to move earth, rocks, and othermaterial from an excavation site. Often, it may be desirable to moveexcavated material from an excavation site to another location remotefrom the excavation site. For example, the material may be loaded ontoan off-highway haulage unit that may transport the materials to a dumpsite. As another example, the material may be excavated by a pull pandrawn behind a tractor, and then hauled, via the pull pan, to the dumpsite. As a further example, a wheel tractor scraper may be used forexcavating, hauling, and dumping the excavated material.

One such machine, a wheel tractor scraper, may be used in an operatingcycle to cut material from one location during a load phase, transportthe cut material to another location during a haul phase, unload the cutmaterial during a dump phase, and return to an excavation site during areturn phase to repeat the operating cycle. The decision to use a wheeltractor scraper, as opposed to some other excavating machine or system,may be based on factors such as the operating cost and the productivityof the machine or system.

The productivity and the cost of operating a machine, or a fleet ofmachines, may be adversely affected by certain factors. For example, anoperator of a wheel tractor scraper may spend too much time in a loadcycle relative to the time required to complete a haul cycle, reducingefficiency. Also, utilizing a particularly long load cycle to fully loador perhaps overload a machine may be efficient in terms of realproductivity and cost for certain haul cycles, but for other haul cyclesmay deteriorate productivity and increase cost by increasing tire slip(increased tire wear), burning more fuel, increasing wear on groundengaging tools, and increasing wear on machine structure and powertraincomponents, for example.

Systems have been designed with a view toward increasing the efficiencyof earthmoving machines, including during the loading phase. Forexample, U.S. Patent Application Publication No. 2016/0289927 to Wang etal. (“the '927 Publication”) describes a bowl-monitoring system with aperception sensor that provides a signal to a controller that isindicative of a view of the bowl of the machine. Based on the signal,the controller determines a level of material in the bowl and providesan indication to an operator of the machine of the current loadingstatus of the bowl.

While the system described in the '927 Publication helps increaseloading efficiency, the system employs three-dimensional perceptionsensors such as LiDAR (Light Detection and Ranging) or a stereo camerato monitor the bowl. These types of perception sensors can be expensive,potentially making it cost-prohibitive to install on a fleet ofmachines, for example. Additionally, three-dimensional image processingmay be computationally expensive, requiring increased computing powerfurther increasing the cost of the '927 Publication's solution relativeto other types of solutions.

The present disclosure is directed to one or more improvements in theexisting technology.

SUMMARY

One aspect of the disclosure is directed to a method for loading of apayload carrier of a machine. The method may include receiving, from acamera on the machine, a two-dimensional image of an interior of thepayload carrier as material is loaded into the payload carrier. Themethod may include filtering the image to identify a contour of theloaded material and determining an area of the contour. The method mayinclude controlling a display device to indicate the determined area.

Another aspect of the disclosure is directed to a camera for assistingloading of a payload carrier of a machine. The camera may include amemory storing instructions and a processor. The processor may beconfigured to execute the instructions to receive a two-dimensionalimage of an interior of the payload carrier as material is loaded intothe payload carrier and filter the image to identify a contour of theloaded material. The processor may be further configured to execute theinstructions to determine an area of the contour and provide a signal tocontrol a display device to indicate the determined area.

Another aspect of this disclosure is directed to a machine. The machinemay have a display device and a camera configured to capture atwo-dimensional image of an interior of a payload carrier of the machinewhen material is loaded into the payload carrier. The machine may alsohave a controller configured to receive the image from the camera and tofilter the image to identify a contour of the loaded material. Thecontroller may be further configured to determine an area of the contourand to provide a signal to control the display device to indicate thedetermined area.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a machine according to an exemplarydisclosed embodiment;

FIG. 2 is a graph of a load growth curve for the machine of FIG. 1;

FIG. 3 is a schematic illustration of an exemplary control system of themachine of FIG. 1.

FIG. 4 is a schematic illustration of an exemplary camera associatedwith the control system of FIG. 3.

FIGS. 5 and 6 are representations of images captured by the camera ofFIG. 4.

FIG. 7 is a representation of an exemplary interface displayed on adisplay device of the machine.

FIG. 8 is a flowchart of an exemplary method for optimizing loading ofthe machine's payload carrier.

FIG. 9 is a flowchart of an exemplary filtering step of the method ofFIG. 8.

DETAILED DESCRIPTION

FIG. 1 diagrammatically illustrates a machine 100 which may be, forexample, a wheel tractor scraper. Machine 100 may be any machine forperforming work on a site using a ground-engaging tool. Machine 100 mayinclude various components or sub-machines such as wheel tractorscrapers, pull-pans, etc.

Machine 100 may include one or more traction devices 102, such as frontand rear wheels, enabling machine 100 to function as a mobile unit. Asuitable power source 104, e.g., a diesel combustion engine, may belocated at the front 106 of machine 100. An additional power source 108,which also may be a diesel engine, may be included at the rear 110 ofmachine 100.

A payload carrier 112 between the front 106 and rear 110 of machine 100may enable machine 100 to transport a quantity of material, such asearth (soil, rock, etc.). On a wheel tractor scraper, payload carrier112 may be a container to receive and hold material for transport andmay sometimes be called a scoop or bowl.

Machine 100 may further include an operator station 114. Operatorstation 114 may include an enclosed or partially-enclosed cab, and mayinclude an operator seat 116, suitable operator control devices 118, adisplay device 120, and/or other components for operating machine 100.

Machine 100 also may include a suitable control system, including acontroller 122, various detectors or sensors, and various actuators foroperating the components of machine 100. For example, machine 100 mayinclude one or more actuators 124, such as hydraulic cylinders, forraising and lowering payload carrier 112. Actuators 124 may lowerpayload carrier 112 so that a ground engaging tool 126, typicallylocated at the lower front edge of payload carrier 112, may penetratematerial to be loaded during a load phase of the machine 100. Actuators124 may also raise payload carrier 112 for transportation of the payloadduring a haul phase of machine 100. Additional actuators may include oneor more actuators 128 to move an ejector 130 during a dump phase and oneor more actuators 132 for controlling an apron 134.

Actuators 132 may move apron 134 from engagement with the front portionof payload carrier 112 to an open position during load and dump phases.Actuators 132, by reverse movement, may also maintain apron 134 in aclosed position engaged with the front portion of the payload carrier112 during a haul phase.

Apron 134 may operate synchronously with ejector 130 during a dumpphase, with actuators 132 moving apron 134 to the open position andactuators 128 moving ejector 130 within payload carrier 112 to assist indumping the payload.

Steering of machine 100 may be facilitated by a steering unit includingone or more actuators 136 located, for example, at a position betweenpayload carrier 112 and the front 106 of machine 100.

As illustrated in FIG. 1, in some embodiments, a load assist unit 138may optionally be associated with payload carrier 112. The exemplaryload assist unit 138 shown in FIG. 1 is representative of various typesof load assist units that may be employed, including, for example, augerunits or elevator units. In FIG. 1, load assist unit 138 is illustratedas an auger 140. It will be understood that the load assist unit 138 mayinclude a plurality of augers, an elevator unit, or other expedientswhich may assist the loading of material into payload carrier 112. Loadassist unit 138 may be driven by a suitable machine actuator, e.g., arotary hydraulic actuator 152.

Machine 100 may include other components to assist the operator inloading and dumping payload carrier 112 and/or to control machine 100autonomously to do so. In the disclosed embodiments, a camera 154 may bepositioned to view the interior of payload carrier 112 to enabledetermination of the amount of material accumulated in payload carrier112. For example, camera 154 may be mounted on a portion of payloadcarrier 112, for example on a mast or stalk, to yield a view of thematerial entering the payload carrier and accumulated therein. In oneembodiment, camera 154 may be a two-dimensional camera such as acategory B (bridge digital) camera.

A machine 100 to which this disclosure applies, for example, a wheeltractor scraper, may operate in cycles that may include load, haul,dump, and return phases. In a given earth- or material-moving operation,such as that carried out by a wheel tractor scraper, machine cycles ofoperation may be affected by various parameters and/or factors which maybe referred to as cycle characteristics. Consideration of cyclecharacteristics during machine operation may enable enhancement,optimization, and/or maximization of machine productivity, along withcontrol of operation costs, through optimization of machine payload.

Cycle characteristics may include, for example, the length of the haulphase of a cycle, the grade to be negotiated by the machine, thecharacter of the ground over which the machine must travel, thecharacter of the machine (i.e., the machine size and manner of loading),the type of material loaded, and machine speed relative to the amount ofpayload.

Another cycle characteristic that may be considered in connection with awheel tractor scraper is a load growth curve. A load growth curve is agraphic representation of the increase in payload volume during machineloading. For a wheel tractor scraper, the load growth curve normally mayindicate that most of the payload volume is loaded early during the loadphase of an operating cycle, with a gradually diminishing increase inpayload later in the load phase.

FIG. 2 graphically illustrates an exemplary load growth curve 200 for amachine 100, such as a wheel tractor scraper. Payload is represented onthe y-axis, and generally may be measured in bank cubic yards (BCY).Load time may be measured on the x-axis, with the unit of time inminutes and/or fractions thereof, for example.

Load growth curve 200 may exhibit a rather steep portion 202 duringinitial stages of loading and may level off, exhibiting a less steepportion 204, as the load phase proceeds. Thus, the bulk of payloadvolume may be accumulated in the part of the load phase corresponding tosteep portion 202, with subsequent increase in payload graduallydiminishing, corresponding to less steep portion 204. Thischaracteristic shape for a load growth curve may be attributed to thefact that, as payload carrier 112 receives more and more material, laterloaded material may be required to lift or force its way throughpreviously loaded material.

As shown in FIG. 2, load growth curve 200 reflects an actual stop time206 and an optimum stop time 208. Actual stop time 206—about 1.2 minutesin this example—may correspond to the time at which an operator inpractice typically actually stops loading payload carrier 112 withmaterial.

Optimum stop time 208—0.6 minutes in this case—may correspond to anoptimum time at which loading should stop to maintain efficient andeffective machine operation. Optimum stop time 208 may correspond to apoint on load growth curve 200 at which steep portion 202 more sharplytransitions to less steep portion 204. In the FIG. 2 example, optimumstop time 208 is about 0.6 minutes. Thus, in this example, an operatormay typically continue loading for about 0.6 minutes after optimum stoptime 208—half of the total loading time. While payload carrier 112accumulated about 18 BCY in the first 0.6 minutes of loading, it onlyaccumulated an additional 3 BCY during additional loading time 210 of0.6 minutes from optimum stop time 208 to actual stop time 206. Thismakes additional loading time 210 an inefficient use of resources,including fuel and time.

It will be appreciated that machines like wheel tractor scrapers mayhave differing load growth curves, depending, for example, on the sizeof the machine, whether the machine is self-loading, whether the machineis push loaded, whether the machine is of the push-pull type, whetherthe machine has an expedient to augment loading (e.g., an elevator orauger), the type of material loaded (e.g., clay, sand, gravel, mixtureof rock and earth, etc.), and/or the size and shape of payload carrier112. The load growth curve for a given machine operating under a givenset of circumstances may be determined empirically, in advance of actualproduction operation of the given machine. This may be accomplished bytest operation and previous field experience, for example.

Controller 122 may include a central processing unit, a suitable memorycomponent, various input/output peripherals, and other componentstypically associated with machine controllers. Controller 122 mayinclude programs, algorithms, data maps, etc., associated with operationof machine 100. Controller 122 may be configured to receive informationfrom multiple sources, such as, for example, one or more of theactuators 124, 128, 132, and 136, camera 154, various sensors ordetectors (e.g., for machine travel direction, ground speed, engineoperation, etc.), as well as input from a machine operator via, forexample, control devices 118. Controller 122 may be suitably located tosend and receive appropriate signals to and from the various sensors,actuators, etc., associated with machine 100. In one embodiment, asshown in FIG. 1, controller 122 may conveniently be located within oradjacent operator station 114. For example, controller 122 may comprisea laptop or mobile computer of the operator. Alternatively, controller122 may comprise a dedicated electronic control module (ECM) or othertype of onboard computer of machine 100. In some embodiments, aspects ofcontroller 122 may be incorporated into camera 154 such that camera 154is “smart camera” configured to perform the disclosed operations ofcontroller 122. In this case, controller 122, or certain aspectsthereof, may be eliminated.

FIG. 3 schematically shows an exemplary control system 300 associatedwith controller 122. Controller 122 may suitably communicate withvarious machine components, for example via conductors. Operator controldevices 118 and display device 120 may enable an operator to manuallysupply signals to controller 122. Display device 120 may, for example,provide an operator with various information to enhance operatorawareness of various machine systems and thereby facilitate maintainingeffective and efficient machine operation. Controller 122 may receivedata input 302 from various sources, including keyboards, a touch screendisplay (which, for example, may be associated with display device 120),computer storage devices, Internet repositories, wireless networks, orother sources of data input known to those skilled in the art.

Controller 122 also may communicate with various machine actuators 304via a machine actuator(s) module 306. Machine actuator module 306 may beconfigured to operate, for example, lift actuator(s) 124, apronactuator(s) 132, ejector actuators(s) 128, bail actuator, steeringactuator(s) 136, load assist actuator(s) 152, or any other actuatorsassociated with machine 100.

Controller 122 may further be configured to communicate with a speedcontrol module 308 to control a mobile speed of machine 100. Speedcontrol module 306 may include, for example, engine speed controlexpedients, throttle control, transmission gear shifting control, etc.

Controller 122 may further be configured to communicate with anautonomous control module 310. Autonomous control module 310 may controlmachine 100 to perform various tasks without any operator input, or withonly a certain amount of operator input. For example, autonomous controlmodule 310 may be configured to operate machine 100 in a loading modefor performing a loading phase at a certain loading location; a haulingmode for performing a hauling phase of hauling the loaded material fromthe loading location to a certain dumping location; a dumping mode forperforming a dumping phase of dumping the material at the dumpinglocation; and/or a return mode for returning machine 100 to the loadinglocation. In response to signals from controller 122, autonomous controlmodule 310 may control machine 100 to perform cycles of the loading,hauling, dumping, and return phases.

Controller 122 may receive input data relevant to cycle characteristics,for example, on an on-going basis. This may enable relatively continualupdating of calculated optimum payloads for machine 100. For example,consistent with the disclosed embodiments, controller 122 may receivedata from camera 154. In some embodiments, controller 122 may employother components (not shown), such as an odometer, inclinometer, wheelslip sensors, another payload sensor (e.g., a scale), and/or variousother sensors, detectors, diagnostic devices, etc. Controller 122 mayuse the data received from these components to gather data relevant tocycle characteristics and control the operations of machine 100.

Consistent with the disclosed embodiments, controller 122 may beconfigured to receive data from camera 154 indicating whether payloadcarrier 112 has been optimally filled with material in accordance withload growth curve 200. In response to this data, controller 122 may beconfigured to provide signals to one or more components of machine 100,such as operator control devices 118, display device 120, machineactuator module 306, speed control module 308, or autonomous controlmodule 310.

For example, in response to receiving a signal from camera 154indicating that payload carrier 112 is optimally filled, controller 122may provide signal(s) to operator control devices 118. Controller 122may provide signals to change a loading indicator light in operatorstation 114 from green (continue loading) to red (stop loading) toindicate to the operator that payload carrier 112 is optimally filled.

Alternatively, or additionally, controller 122 may be configured toprovide signal(s) to display device 120 indicating that payload carrier112 is optimally filled. Display device 120 may, in turn, may provide avisual indication on the display letting the operator know that payloadcarrier 112 is optimally filled.

Alternatively, or additionally, controller 122 may be configured toprovide signal(s) to machine actuator module 306 indicating that payloadcarrier 112 is optimally filled. Machine actuator module 306, in turn,may provide signals to actuate one or more actuators. For example,machine 306 may provide one or more signals to: (1) lift actuator 124 toraise payload carrier 112; (2) apron actuator 132 to move apron 134 froman open position to a closed position engaged with the front portion ofpayload carrier 112; (3) ejector actuator 128 to move ejector 130 withinpayload carrier 112, such as to dump the payload or stow ejector 130 forthe hauling phase; (4) a bail actuator 312 to manipulate a bail at thefront 106 of machine 100; (5) steering actuator 136 to change the anglebetween the front 106 and rear 110 sections of machine 100; or (6) loadassist actuator 152 to stow a load assist unit for the haul phase.

Alternatively, or additionally, controller 122 may be configured toprovide signal(s) to speed control module 308 indicating that payloadcarrier 112 is optimally filled. In response to the signal(s), speedcontrol module 308 may be configured to reduce the speed of machine 100or stop machine 100, to reduce throttle or the speed of power source(s)104, 108, etc.

Alternatively, or additionally, controller 122 may be configured toprovide signal(s) to autonomous control module 310 indicating thatpayload carrier 112 is optimally filled. In response to the signal(s),autonomous control module 310 may, for example, change the currentoperating mode of machine 100 from the loading mode to the haul mode orperform other functions to complete the loading phase.

FIG. 4 shows an exemplary schematic representation of camera 154. Camera154 may have computing components for digital cameras. For example,camera 154 may have memory 400, data storage 402, a communication unit404, a lens unit 406, a processor 408 configured to execute a payloadoptimization algorithm 410.

Memory 400 may include temporary data storage such as RAM. Data storage402 may include persistent storage such as ROM, Flash, solid state, orother type of data storage known in the art.

Communication unit 404 may be configured to communicate with externalcomponents, such as controller 122. Communication unit 404 may includeexample, USB, Firewire, Bluetooth, Wi-Fi, CAN bus, Ethernet or otherelectronic communication interfaces known in the art for interconnectingcomputing devices. Under the command of processor 408, communicationunit 404 may intermittently or continually send data signals tocontroller, including signals indicating whether payload carrier 112 hasbeen determined to be optimally loaded. In some embodiments,communication unit 404 may also stream live video data to controller 122for display or processing.

Consistent with the disclosed embodiments, lens unit 406 may compriseany two-dimensional lens system known in the art for digital cameras.For example, lens unit 406 may embody a digital single-lens reflex(DSLR) system including a lens (e.g., 35 mm lens) and a two-dimensionalimage sensor, such as CCD or CMOS image sensor. It is to be appreciatedthat lens unit 406 may be the same type of lens unit used inconventional digital cameras or smartphones.

Lens unit 406 may output to processor 408 images in the form of acontinuous or intermittent data stream containing color values for eachpixel of the two-dimensional image sensor (e.g., a color filter array).Thus, the data stream may contain two-dimensional image information.Camera 154 may be positioned so that lens unit 406 views the interior ofpayload carrier 112 without obstruction and the output data stream thuscontains two-dimensional image information for the interior of payloadcarrier 112.

Processor 408 may be an image processor known in the art for digitalcameras. For example, processor 408 may be a digital signal processor(DSP) configured to perform various types of imaging processing, such asBayer transformation, demosaicing, noise reduction, imaging sharpening,edge-detection, or coordinate system transformation.

Payload optimization algorithm 410 may comprise computer programinstructions installed on processor 408 and/or stored in memory 400 orstorage 402 for execution by processor 408. Payload optimizationalgorithm 410, executed by processor 408, may be configured to processthe two-dimensional digital image data received from lens unit 406 todetermine when payload carrier 112 is optimally loaded. To do so,algorithm 410 may transform the two-dimensional data received from lensunit 406 from a first coordinate system associated with camera 154 to asecond, reference coordinate system used to determine the amount ofmaterial in payload carrier 112.

FIG. 5 shows a sequence of images 500-504 illustrating how algorithm 410may transform the two-dimensional image data from lens unit 406. It willbe appreciated that camera 154 may need to be placed so that it does notinterfere with operations of machine 100 yet still has a view of theinterior of payload carrier 112. For example, payload carrier 112 mayhave a generally rectangular shape, and camera 154 may sit on a mast atthe rear right of payload carrier 112. Thus, camera 154 may have a viewfrom diagonally across and above payload carrier 112. Accordingly,camera 154 may produce an image 500 of payload carrier 112 similar thatshown in FIG. 5.

Additionally, in some embodiments, camera 154 may have a fisheye lens orother wide-angle lens to capture the entire payload carrier 112 from aclose position. Thus, in image 500 generated by camera 154, top edges506 of payload carrier 112 may define axes of a coordinate system 508(X″, Y″) distorted relative to reality. For example, coordinate system508 may bulge outward from “barrel” distortion induced by a wide-anglelens. It will be appreciated that the pixel locations (X″, Y″) of image500 inside coordinate system 508 may thus be visually distorted relativeto reality.

Using image distortion correction techniques and the specifications ofcamera 154, algorithm 410 may be configured to correct image 500 toremove the distortion, producing undistorted image 502. For example,before mounting camera 154 to machine 100, camera 154 may be calibratedusing an open-source “checkerboard” technique that outputs calibrationparameters required to produce rectangular checkerboard of apredetermined size from the distorted image 500. Algorithm 410 may beconfigured to apply the calibration parameters to image 500 to produceimage 502.

In undistorted image 502, top edges 506 of payload carrier 112 maydefine an undistorted coordinate system 510 (X′, Y′) in which the X′-and Y′-axes extend in straight lines. Algorithm 410 may be configured totranslate the pixel locations (X″, Y″) in coordinate system 508 to theircorresponding locations (X′, Y′) in coordinate system 510.

As can be seen in FIG. 5, in image 510, payload carrier 112 appearsrotated about 45 degrees relative to horizontal. Additionally, becausethe view from image 502 remains from diagonally across and above payloadcarrier 112, the X′ and Y′ axes intersect one another at an obtuse angle(i.e., greater than 90 degrees).

To produce an image 504 by which the amount of material in payloadcarrier 112 may be determined, algorithm 410 may be configured to rotateand translate the pixel locations (X′, Y′) in coordinate system 510 tocorresponding locations (X, Y) in a working coordinate system 512. Incoordinate system 512, the X- and Y-axes intersect one another at aright angle (i.e., 90 degrees) as in a standard cartesian coordinatesystem. To accomplish this, algorithm 410 may use rotational andtranslational transformation techniques. For example, algorithm 410 maybe configured to apply perspective transformation (e.g., an OpenCValgorithm) so that the corners of payload carrier 402 form a rectangle,transforming image 502 to top-down image 504. Image 504 may correspondto a top-down view of payload carrier 112 from vertically above it,i.e., looking straight down at payload carrier 112 from above.

FIG. 6 shows sequential images 600-604 of payload carrier 112 as it maybe filled with material during loading, from the top-down perspective ofimage 504. In image 600, loading of payload carrier 112 has just begun,so payload carrier 112 may only contain a small amount of material 606.Material 606 may define a contour 608, around its edges or perimeter,enclosing an area 610. It will be appreciated that area 610 may be avertical cross-sectional area of material 606 as it sits in payloadcarrier 112 (i.e., the area of the “footprint” of material 606) becauseof the top-down perspective of image 504.

As payload carrier 112 fills further with material 606, contour 608 mayexpand, enclosing a larger and larger area 610. Image 602 shows payloadcarrier 112 when partially filled (e.g., 50%) with material 606, andimage 604 shows payload carrier 112 when optimally filled (e.g., 85%)with material 606. In this description, “optimally” filled may refer toa payload volume corresponding to a desired optimum stop time 208 onload growth curve 200, at which steep portion 202 transitions to lesssteep portion 204.

The optimum volume and corresponding desired optimum stop time 208 maybe determined in different ways. For example, they may be determinedempirically through field testing of machine 100. Alternatively, anoperator or other person knowledgeable about the performancecharacteristics of machine 100 may select the optimum volume/stop time208 based on experience. The optimum volume/stop time 208 may also bedetermined mathematically by selecting a point on load growth curve 200at which the slope of load growth curve 200 reaches a certain desiredthreshold (e.g., 30%).

The area 610 of material 606 may correspond generally to the actualvolume of material 606 in payload carrier 112. For example, certainmaterials are known to rest at certain angles of repose. Thus, ifmaterial 606 has a certain area 610, material 606 may have certaincorresponding height. Likewise, as area 610 expands by a known amount,it may be assumed that the height of material 606 also grows by acorresponding known amount. This may allow a relative volume of thematerial to be calculated based on the cross-sectional area 610.Additionally, an actual value for the volume of the material may becomputed from the relative volume based on the known dimensions ofpayload carrier 112. Accordingly, in the disclosed embodiments, area 610of material 608 in payload carrier 112 may be used as a substitute orsurrogate for volume.

In the examples shown in FIG. 6, it may be assumed that the area 610 ofmaterial 606 shown in image 504 corresponds to the optimum loadingvolume for payload carrier 112 (e.g., 85%) based on load growth curve200. Thus, based on the above discussion, it may be desirable to stoploading payload carrier 112 when area 610 reaches the threshold areashown in image 504.

Algorithm 410 may be configured to calculate area 610 encompassed bycontour 608 using imaging processing techniques. In one embodiment,algorithm 410 may be configured to apply a feature-detection algorithm(e.g., an OpenCV algorithm), which may output pixel values (X, Y) offeatures in image 504. For example, sharp angles (e.g., >60 degrees) maybe identified as features, such as the corners of payload carrier 112 orthe corners of material 606.

Algorithm 410 may be configured to then filter/remove the coordinates(X, Y) corresponding to features other than material 606. In oneembodiment, algorithm 410 may be configured to use motion detection. Forexample, algorithm 410 may apply the Lucas-Kanade Optical flow techniqueas implemented in OpenCV, using as input the pixel values (X, Y) of thedetected features, the current image 504, and the previous image 504. Asoutput, algorithm 410 may be configured to provide motion vectors forthe pixel values (X, Y) corresponding each of detected featurescontaining. The motion vectors may include the magnitude of the motionand the direction of the motion of the features between the two images504.

Algorithm 410 may be configured to remove the pixel values (X, Y)corresponding to features with motion vectors whose magnitudes are abovea threshold. For example, algorithm 410 may be configured to determinethe mean magnitude of the motion vectors and the standard deviation ofthe motion vectors. Algorithm 410 may then remove the features withpixel values (X, Y) having motion vectors with a magnitude greater thanthe mean by a certain threshold, such as two standard deviations. Asexplained below, following this process, only the coordinates (X, Y)corresponding to material 606 may remain.

Algorithm 410 may be configured to determine area 610 of contour 608.For example, algorithm 410 may be configured to count the number ofpixels remaining following the filtering, which is area 610 as measuredin square pixels. Algorithm 410 may also be configured to determine apercentage fill factor for the payload carrier 112. The area of payloadcarrier 112 may be the total number of pixels within the four corners ofpayload carrier 112, which may be determined in advance and/or fixed. Todetermine the percentage fill factor, algorithm 410 may be configured todivide the counted number of pixels within contour 608 by the totalnumber of pixels within the four corners of payload carrier 112.

Algorithm 410 may be configured to use other techniques to determinearea 610 of contour 608, if desired. For example, algorithm 410 mayapply edge detection to identify the pixel values (X, Y) correspondingto contour 608. In one embodiment, algorithm 410 may be configured toidentify pixels in image 504 where the color values transition from thecolor of material 606 (e.g., brown, black, or dark gray in a grayscaleimage) to the color of payload carrier 200 (e.g., yellow or light grayin a grayscale image). Algorithm 410 may deem these identified pixels ascontour 608. Next, algorithm 410 may be configured to identify allpixels in image 504 that fall within contour 608. Algorithm 410 may beconfigured to count the total number of pixels making up contour 608 andwithin the region encompassed by contour 608. Algorithm 410 may bedetermined to determine a percentage fill factor by dividing the totalnumber of pixels of contour 608 by the total number of pixels inside thefour corners of payload carrier 112.

As shown in FIG. 7, algorithm 410 may be configured to provide signalsto controller 122 to control display device 120 to display a payloadcarrier status interface 700. Interface 700 may have one or more userinterface elements allowing the operator to provide or controlinformation about the status of payload carrier 112. For example,interface 700 may include a video feed window 702 displaying a livevideo feed from camera 154, corresponding to image 500. Interface 700may further include a payload growth curve 704 illustrating the currentloading status of payload carrier 112.

Interface 700 may further have a fill factor indicator 706 indicatingthe percentage fill factor of payload carrier 112, i.e., the percentageof payload carrier 112 that is filled with material 606. Interface 700may also be configured to provide a notification that payload carrier112 is optimally filled upon receiving a corresponding signal fromcamera 154. For example, interface 700 may be configured to display a“Stop Loading” message upon receipt of such a signal so that theoperator knows to stop the current loading phase.

Interface 700 may further include an option 708 to set the top corners710 and bottom corners 712 of payload carrier 112. For example, uponselecting option 708, the operator or a technician may use a mouse, atouch screen of display device 120, or other user input device to setthe corners 710, 712. Once set, algorithm 410 may be configured to usecorners 710, 712 to translate image 500 to image 502, as explainedabove.

FIG. 8 shows an exemplary method 800 for optimizing loading of payloadcarrier 112 during operation of machine. Method 800 may be performed byalgorithm 410 when executed by processor 408. The steps of method 800need not necessarily be performed in the order shown in FIG. 8 and maybe performed in different orders consistent with the disclosedembodiments.

In step 802, algorithm 410 may receive a live video feed correspondingto image 500 from lens unit 406. The video feed may be displayed inwindow 702 of interface 700.

In step 803, algorithm 410 may remove distortion from an image 500 inthe feed as explained above. For example, in an embodiment using awide-angle lens, algorithm 410 may remove barrel distortion. Thus, forexample, step 803 may convert pixel values (X″, Y″) in coordinate system508 to corresponding pixel values (X′, Y′) in coordinate system 510.

In step 804, algorithm 410 may transform image 500 from coordinatesystem 510 to coordinate system 512 of image 504, as discussed abovewith respect to FIG. 5. For example, previously the operator may haveused option 708 to select top corners 710 and bottom corners 712 ofpayload carrier 112. Algorithm 410 may use the pixel values (X′, Y′) forcorners 710 and/or 712 of payload carrier 112 to translate all pixelvalues (X′, Y′) in coordinate system 510 to their corresponding values(X, Y) in coordinate system 512, as explained.

In step 806, algorithm 410 may filter the transformed image 504 toremove pixel values corresponding to features other than material 606,as explained above. FIG. 9 shows a method for an exemplary step 806.

In step 900, algorithm 410 may detect features in image 504. Forexample, as explained above, algorithm 410 may apply a feature-detectionprocess (e.g., an OpenCV process). The feature-detection process mayoutput pixel values (X, Y) of any identified features in image 504, suchas the corners of payload carrier 112 and the edges or corners ofmaterial 606 in payload carrier 112.

In step 902, algorithm 410 may calculate motion vectors for the featuresidentified in step 900. For example, as explained above, algorithm 410may apply a motion-detection technique such as the Lucas-Kanade Opticalflow technique. Using the input the pixel values (X, Y) of the featuresdetected in step 900, a prior image 504, and the current image 504 asinput, algorithm 410 may calculate motion vectors for each of thedetected features. Each motion vector may have a value for a magnitudeof the feature's motion between the prior image 504 and the currentimage 504. Additionally, in some embodiments, each vector may have avalue for the direction of the motion between the prior image 504 andthe current image 504. For example, a motion vector for a given featuremay be 10 pixels, 45 degrees.

In step 904, algorithm 410 may calculate a mean magnitude of the motionvectors calculated in step 902 (e.g., in pixels). Algorithm 410 mayadditional calculate a standard deviation of the magnitudes (e.g., inpixels).

In step 906, algorithm 410 may determine whether the magnitude of eachvector calculated in step 902 is greater than a threshold (e.g., 12pixels). In one embodiment, the threshold may be the mean magnitudecalculated in step 904 plus a certain number X of standard deviations.This is because X=2 standard deviations may provide a thresholddistinguishing features corresponding to material 606 from other movingfeatures. For example, pixel values (X, Y) corresponding to material 606may move by certain magnitudes, from one image 504 to the next, aspayload carrier 112 fills with material 606 and contour 608 expands.Thus, the threshold may be selected so that the various magnitudes bywhich material 606 typically moves between images 504 fall within thethreshold. In the X=2 example above, features with motion vectors whosemagnitudes are less than or equal to the mean magnitude plus twostandard deviations may be known to correspond to moving or stationarymaterial 606. On the other hand, features whose motion vectors havemagnitudes greater than the mean magnitude by more than two standarddeviations may be known to be moving too quickly to be material 606.They may be, for example, the surrounding environment, shadows,momentary obstructions to the view of camera 154, or other features thatare not material 606.

If the result of step 906 is no, the pixel values corresponding to thefeatures have been determined to correspond to material 606. Thus, instep 908, the pixel values may be kept for the area calculation. Forexample, algorithm 410 may store those pixel values in memory 400 ofcamera 154 for further processing.

If the result of step 906 is yes, the pixel values corresponding to thefeatures have been determined not to correspond to material 606. Thus,in step 910, the pixel values may be discarded and not used in the areacalculation.

Returning to FIG. 8, in step 808, algorithm 410 may determine area 610of contour 608. For example, algorithm 410 may count the number ofpixels kept in step 908 and thus remaining after the filtering of FIG.9. This number of pixels may correspond to area 610, in square pixels.Additionally, algorithm 410 may determine the percentage fill factor bydividing the number of counted pixels by the total number of pixelswithin the four corners of payload carrier 112 in image 504, which maybe determined in advance and/or fixed. Algorithm 410 may providesignal(s) to controller 122 so that controller 122 controls displaydevice 120 to indicate the calculated fill factor on indicator 706within interface 700.

In step 810, algorithm 410 may determine whether the area 610 determinedin step 808 is equal to or greater than a threshold. For example,algorithm 410 may determine whether the fill factor determined in step808 is greater than or equal to a threshold percentage (e.g., 85%). Asdiscussed above, the threshold percentage may be predetermined tocorrespond to a desired optimum loading volume of payload carrier 112(e.g., 85%) relative to the total loading capacity.

If the result of step 810 is “no,” algorithm 410 may return to step 802,and algorithm may repeat steps 802-812 until area 610 reaches thethreshold, meaning that payload carrier 112 has been loaded to theoptimum volume.

If the result of step 810 is “yes,” algorithm 410 may notify controller122 that payload carrier 122 has been determined to be optimally loaded,in step 812. For example, as discussed above, processor 408 may transmita signal via communication unit 404 to controller 122 indicating thatpayload carrier 122 has been optimally loaded.

As explained above, controller 122 may take one or more actions based onthis notification, including any combination of, for example:

-   -   Provide signals to change a loading indicator light in operator        station 114 from green (continue loading) to red (stop loading)        so the operator knows payload carrier 112 is optimally filled.        The operator may then manually control machine 100 to stop        loading payload carrier.    -   Provide signal(s) to display device 120 indicating that payload        carrier 112 is optimally filled. Display device 120 may, in        turn, may provide a visual indication on the display (e.g., a        “Stop Loading” message) letting the operator know that payload        carrier 112 is optimally filled. The operator may then manually        control machine 100 to stop loading payload carrier.    -   Provide signal(s) to machine actuator module 306 indicating that        payload carrier 112 is optimally filled. Machine actuator module        306, in turn, may provide signals to actuate actuators to        complete the loading phase. For example, machine 306 may provide        one or more signals to: (1) lift actuator 124 to raise payload        carrier 112; (2) apron actuator 132 to move apron 134 from an        open position to a closed position engaged with the front        portion of payload carrier 112; (3) ejector actuator 128 to move        ejector 130 within payload carrier 112, such as to dump the        payload or stow ejector 130 for the hauling phase; (4) a bail        actuator 312 to manipulate a bail at the front 106 of machine        100; (5) steering actuator 136 to change the angle between the        front 106 and rear 110 sections of machine 100; or (6) load        assist actuator 152 to stow a load assist unit for the haul        phase.    -   Provide signal(s) to speed control module 308 indicating that        payload carrier 112 is optimally filled. In response to the        signal(s), speed control module 308 may be configured to bring        the loading phase to an end by reducing the speed of machine        100, stopping machine 100, reducing the throttle or the speed of        power source(s) 104, 108, etc.    -   Provide signal(s) to autonomous control module 310 indicating        that payload carrier 112 is optimally filled. In response to the        signal(s), autonomous control module 310 may, for example,        change the current operating mode of machine 100 from the        loading mode to the haul mode or perform other functions to        complete the loading phase.

INDUSTRIAL APPLICABILITY

The disclosed embodiments may apply to work machines, such as, forexample, wheel tractor scrapers, which may operate in cycles that mayinclude load, haul, dump, and return phases. It is beneficial tocomplete these cycles as efficiently as possible by eliminating wastedtime and resources such as person-hours and fuel. Efficiency may beincreased by accounting for cycle characteristics—one of which is theload growth curve of the machine discussed herein.

Specifically, time and resources may be saved by accounting for loadgrowth curve 200 for machine 100. By ensuring that, for each load cycle,operators do not continue loading payload carrier 112 beyond the optimumvolume, more cycles may be completed in less time using less fuel.

The disclosed embodiments may provide a relatively inexpensive buteffective technique to notify the operator to stop loading—or toautonomously control machine 100 to do so—once the optimum fill volumeis reached. While payload volume can be determined withthree-dimensional imaging systems such as LiDAR or stereo cameras, suchsystems are expensive. Thus, it may be cost-prohibitive to usethree-dimensional payload imaging systems, for example, on a fleet ofmachines.

By contrast, the disclosed embodiments may instead apply an inexpensivetwo-dimensional camera 154, such as the types used on conventional,mass-produced smartphones or digital cameras. Volume may not be directlycalculated from a two-dimensional image, but the disclosed embodimentsmay use an area 610 of the payload material 606 as a surrogate toindirectly determine when the optimum payload volume has been reached.This advantageously allows the use of relatively inexpensivetwo-dimensional cameras instead of more expensive three-dimensionalcameras. Additionally, two-dimensional image processing typically mayrequire fewer computing resources than three-dimensional imageprocessing. Thus, the disclosed embodiments may reduce cost by requiringfewer computing resources.

Additionally, the disclosed embodiments may enable the operator tobetter focus on safely driving the machine. By providing a notification(e.g., indicator light or indication on the display) when the payloadcarrier is optimally filled, the operator may not need to turn around toview the payload carrier to determine whether it is full, as in the caseof conventional wheel tractor scrapers.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed payloadoverload control system without departing from the scope of thedisclosure. Other embodiments will be apparent to those skilled in theart from consideration of the specification and practice of theembodiments disclosed herein. For example, this disclosure may describeembodiments in which camera 154 is “smart” and configured to performalgorithm 410. This may allow camera 154 to be offered as a standaloneunit (e.g., kit) for retrofitting an older machine that does nototherwise have the disclosed payload optimization functionality.However, machines may also be equipped with this functionality as astandard or optional feature. For example, this disclosure also includesusing an ordinary camera instead of a “smart” camera. In suchembodiments, one or more of the functions of camera 154, including oneor more functions of algorithm 410, may be embedded in controller 122instead of the camera, and the camera need only capture the images ofthe payload carrier and provide them to the controller. Thus, it isintended that the specification and examples be considered as exemplaryonly, with a true scope of the disclosure being indicated by thefollowing claims and equivalents thereof

What is claimed is:
 1. A method for loading of a payload carrier of amachine, the method comprising: receiving, from a camera on the machine,a two-dimensional image of an interior of the payload carrier asmaterial is loaded into the payload carrier; filtering the image toidentify a contour of the loaded material; determining an area of thecontour of the loaded material; determining that the area is equal to orgreater than to a threshold; based at least in part on determining thatthe area is equal to or greater than the threshold, performing one ormore of the following: controlling a display device in an operatorstation of the machine to indicate that the payload carrier of themachine is full; activating an indicator light in the operator stationto indicate to stop loading the payload carrier; providing a visualindication on the display device to indicate to stop loading the payloadcarrier; controlling the machine to raise the payload carrier from aground surface; controlling the machine to move an apron of the payloadcarrier from an open position to a closed position to prevent furtherloading of the payload carrier; controlling the machine to reduce speedor to stop; and controlling the machine to change a current operatingmode of the machine to an autonomous haul mode.
 2. The method of claim1, wherein the threshold is a percentage of a total area of the payloadcarrier.
 3. The method of claim 1, wherein determining an area includescounting a number of pixels within the contour.
 4. The method of claim1, wherein filtering the image includes: identifying pixelscorresponding to features in the image; calculating motion vectors forthe identified pixels; and removing pixels corresponding to motionvectors having a magnitude greater than a threshold.
 5. The method ofclaim 1, further comprising removing, from the image, distortion causedby a wide-angle lens before determining the area.
 6. A camera forassisting loading of a payload carrier of a machine, the cameracomprising: a memory storing instructions; and a processor configured toexecute the instructions to: receive a two-dimensional image of aninterior of the payload carrier as material is loaded into the payloadcarrier; filter the image to identify a contour of the loaded material;determine an area of the contour; determine that the area is equal to orgreater than to a threshold; and based at least in part on determiningthat the area is equal to or greater than the threshold, provide asignal to a controller associated with the machine, the signal causingperformance of one or more of the following: control a display device inan operator station of the machine to indicate that the payload carrieris full; activate an indicator light in the operator station to indicateto stop loading the payload carrier; provide a visual indication on thedisplay device to indicate to stop loading the payload carrier; controlthe machine to raise the payload carrier from a ground surface; controlthe machine to move an apron of the payload carrier from an openposition to a closed position to prevent further loading of the payloadcarrier; control the machine to reduce speed or to stop; and control themachine to change a current operating mode of the machine to anautonomous haul mode.
 7. The camera of claim 6, wherein the threshold isa percentage of a total area of the payload carrier.
 8. The camera ofclaim 6, wherein to determine an area of the contour, the one or moreprocessors are further configured to execute the instructions to count anumber of pixels within the contour.
 9. The camera of claim 6, whereinto filter the image, the one or more processors are further configuredto execute the instructions to: identify pixels corresponding tofeatures in the image; calculate motion vectors for the identifiedpixels; and remove pixels corresponding to motion vectors having amagnitude greater than a threshold.
 10. The camera of claim 6, whereinthe one or more processors are further configured to execute theinstructions to remove, from the image, distortion caused by awide-angle lens before determining the area.
 11. A machine, comprising:a display device; a camera configured to capture a two-dimensional imageof an interior of a payload carrier of the machine when material isloaded into the payload carrier; and a controller configured to: receivethe image from the camera; filter the image to identify a contour of theloaded material; determine an area of the contour; determine that thearea is equal to or greater than to a threshold; and in response todetermining that the area is equal to or greater than the threshold,provide a signal causing performance of one or more of the following:control the display device to indicate that the payload carrier is full;activate an indicator light in an operator station of the machine toindicate to stop loading the payload carrier; provide a visualindication on the display device to indicate to stop loading the payloadcarrier; control the machine to raise the payload carrier from a groundsurface: control the machine to move an apron of the payload carrierfrom an open position to a closed position to prevent further loading ofthe payload carrier; control the machine to reduce speed or to stop; andcontrol the machine to change a current operating mode of the machine toan autonomous haul mode.
 12. The machine of claim 11, wherein thethreshold is a percentage of a total area of the payload carrier. 13.The machine of claim 11, wherein the machine is a wheel tractor scraper.14. The machine of claim 11, wherein to determine an area of thecontour, the controller is further configured to execute theinstructions to count a number of pixels within the contour.
 15. Themachine of claim 11, wherein to filter the image, the controller isconfigured to execute the instructions to: identify pixels correspondingto features in the image; calculate motion vectors for the identifiedpixels; and remove pixels corresponding to motion vectors having amagnitude greater than a threshold.
 16. The machine of claim 11, whereinthe controller is further configured to execute the instructions toremove, from the image, distortion caused by a wide-angle lens beforedetermining the area.